Schwarz, Wallace, and Rissanen: Intertwining Themes in Theories of Model Selection

Investigators interested in model order estimation have tended to divide themselves into widely separated camps; this survey of the contributions of Schwarz, Wallace, Rissanen, and their coworkers attempts to build bridges between the various viewpoints, illuminating connections which may have previously gone unnoticed and clarifying misconceptions which seem to have propagated in the applied literature. Our tour begins with Schwarz’s approximation of Bayesian integrals via Laplace’s method. We then introduce the concepts underlying Rissanen’s minimum description length principle via a Bayesian scenario with a known prior; this provides the groundwork for understanding his more complex non-Bayesian MDL which employs a “universal” encoding of the integers. Rissanen’s method of parameter truncation is contrasted with that employed in various versions of Wallace’s minimum message length criteria. Rissanen’s more recent notion of stochastic complexity is outlined in terms of Bernardo’s information-theoretic derivation of the Jeffreys prior.